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Abstract: The current study offers a formative analysis of the impact of learning objects in 
middle school mathematics and science classrooms. Five reliable and valid measure of 
effectiveness were used to examine the impact of learning objects from the perspective of 262 
students and 8 teachers (14 classrooms) in science or mathematics. The results indicate that 
teachers typically spend 1-2 hours finding and preparing for learning-object based lesson plans 
that focus on the review of previous concepts. Both teachers and students are positive about 
the learning benefits, quality, and engagement value of learning objects, although teachers are 
more positive than students. Student performance increased significantly, over 40%, when 
learning objects were used in conjunction with a variety of teaching strategies. It is reasonable 
to conclude that learning objects have potential as a teaching tool in a middle school 
environment. 

L'impacte des objets d'apprentissage dans les classes de mathematique et de 
sciences a I'ecole intermediate : une analyse formative 

Resume : Cette etude presente une analyse formative de l'impacte des objets d'apprentissage 
dans les classes de mathematique et de sciences a I'ecole intermediate. Cinq mesures de 
rendement fiables et valides ont ete exploitees pour examiner I'effet des objets 
d'apprentissage selon 262 eleves et 8 enseignants (414 classes) en science ou mathematiques. 
Les resultats indiquent que les enseignants passent typiquement 1-2 heures pour trouver des 
objets d'apprentissage et preparer les legons associees qui seraient centrees sur la revue de 
concepts deja vus en classe. Quoique les enseignants aient repondu de fagon plus positive que 
les eleves, les deux groupes ont repondu positivement quant aux avantages au niveau de 
I'apprentissage, a la qualite ainsi qu'a la valeur motivationnelle des objets d'apprentissage. Le 
rendement des eleves aurait aussi augmente de fagon significative, plus de 40%, quand les 
objets d'apprentissage ont ete exploites avec une variete de strategies d'enseignement. II 
serait done raisonnable de conclure que les objets d'apprentissage ont un potentiel comme 
outils d'enseignement a I'ecole intermediaire. 

Overview 

Until recently, the vast majority of use and research with respect to learning objects has been 
in the domain of higher education (e.g., Haughey & Muirhead, 2005; Kay & Knaack, in press). 



Increased presence of learning objects in K-6 (e.g., Bower, 2005; Brush & Saye, 2001; Clarke & 
Bowe, 2006a, 2006b; Freebody, Muspratt, & McRae, 2007; Kay & Knaack, 2005; Liu & Bera, 
2005; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005) indicates that there is a need for more 
research on learning objects in middle school environments. Research on the impact, 
effectiveness, and usefulness of learning objects in the K-6 domain is limited, partially because 
comprehensive, theoretically-based, reliable, and valid evaluation tools are scarce. The 
purpose of the current study is to examine the impact of learning objects in middle school 
classrooms. 

Literature Review 

Definition and Benefits of Learning Objects 

Learning objects (LO) are defined in this paper as "interactive web-based tools that support 
learning by enhancing, amplifying, and guiding the cognitive processes of learners". This 
definition was arrived at based on a aggregate of previous definitions (Agostinho, Bennett, 
Lockyer & Harper, 2004; Butson, 2003; McGreal, 2004; Parrish, 2004; Wiley, et al. 2004). 

Learning objects offer a number of advantages for educators and students including 
accessibility (Wiley, 2000), ease of use (e.g., Gadanidis, Gadanidis, & Schindler, 2003; Sedig & 
Liang, 2006), reusability (e.g., Agostinho et al., 2004; Duval, Hodgins, Rehak & Robson, 2004; 
Rehak & Mason, 2003), interactivity (e.g., Gadanidis, et al., 2003; Sedig & Liang, 2006) and 
visual supports (e.g., Gadanidis, et al., 2003; Sedig & Liang, 2006). 

With respect to enhancing learning, many learning objects are interactive tools that support 
exploration, investigation, constructing solutions, and manipulating parameters instead of 
memorizing and retaining a series of facts. The success of this constructivist based model is 
well documented (e.g., Albanese & Mitchell, 1993; Bruner, 1983, 1986; Carroll, 1990; Collins, 
Brown, & Newman, 1989; Vygotsky, 1978). In addition, a number of learning objects have a 
graphical component that helps make abstract concepts more concrete (Gadanidis et al., 
2003). Furthermore, certain learning objects allow students to explore higher level concepts by 
reducing cognitive load. They act as perceptual and cognitive supports, permitting students to 
examine more complex and interesting relationships (Sedig & Liang, 2006). Finally, learning 
objects are adaptive, allowing users to have a certain degree of control over their learning 
environments, particularly when they are learning and for how long. 

In spite of this list of potential benefits, relatively little systematic research has been done 
examining the actual use and impact of learning objects in middle school classrooms. 

Impact of Learning Objects in Elementary and Middle Schools 

An extensive review of the literature on learning objects in the past 10 years uncovered nine 
studies focussing on elementary (Bower, 2005; Clarke & Bowe, 2006a, 2006b; Reimer & Moyer, 
2005) and middle school (Akpinar & Bal, 2006; Bower, 2005; Freebody et al., 2007; Kong & 
Kwok, 2005; Liu & Bera, 2005) grade levels. Note that with the exception of three detailed, 
comprehensive reports from the Learning Federation (Clarke & Bowe, 2006a, 2006b; Freebody 
et al., 2007) these studies were selected from peer-reviewed journals. The papers go back no 
further than 2005, indicating that research in this domain is fairly new. 



Context. The context of use varied with respect to investigations of learning objects. In terms 
of time spent using learning objects, some studies observed students use these tools for 40-60 
minutes (Akinpar & Bal, 2006; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005). Other studies, 
investigated the use of learning objects over several weeks (Clarke & Bowe, 2006a, 2006b; 
Freebody et al., 2007; Liu & Bera, 2005). With respect to the number of learning objects 
evaluated, some papers focussed on a single learning object, while others looked at multiple 
learning objects (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007; Nurmi & Jaakkola, 
2006). Finally, regarding implementation strategies, a majority of teachers played a facilitating 
role, allowing students to explore the learning object on their own (Liu & Bera, 2005; Nurmi & 
Jaakola, 2006; Reimer & Moyer, 2005), although other researchers (Clarke & Bowe , 2006a, 
2006b; Freebody et al., 2007) reported the use of numerous strategies over an extended 
period of time including large group discussions, worksheets, collaborative learning, writing 
reflective comments, and integration with other subject areas. 

Teacher perspective. Three reports looked specifically at teacher attitudes toward the use of 
learning objects in the classroom (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007). 
Qualitative and survey data collected suggested that instructors valued the following 
characteristics: immediate feedback, the ability to replay and redo tasks for both enjoyment 
and mastery, and the motivational qualities. To date, no peer-reviewed journal articles could 
be found that examined middle school teacher's attitudes toward of learning objects. 

Student perspective. Four studies examined student attitudes toward learning objects 
(Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007; Reimer & Moyer, 2005). Students 
reported liking learning objects because they were (a) fun and enjoyable, (b) easy to control 
with respect to the pace of learning, (c) provided timely feedback, (d) consisted of a number 
multimedia tools, and (e) helped them learn (Clarke & Bowe, 2006a, 2006b; Freebody et al., 
2007; Reimer & Moyer, 2005). 

Student performance. In contrast to research presented for secondary and higher education 
domains (Kay & Knaack, 2005, 2007a, 2007b), the majority of studies done on learning objects 
and K-6 classrooms have focussed on some form of student performance (Akpinar & Bal, 2006; 
Bower, 2005; Freebody et al., 2007; Kong & Kwok, 2005, Liu & Bera, 2005; Nurmi & Jaakkola, 
2006; Reimer & Moyer, 2005). In all seven studies, students who used learning objects showed 
significant improvement on various performance measures. Several contextual details, though, 
are worth noting. 

Nurmi & Jaakkola (2006) noted that learning performance was dependent on the type of 
learning object and how it was used. Students working with drill-and-practice learning objects 
tended to be less focussed on learning and more focussed on competing with their peers. 
Students involved in a mixed learning object / laboratory lesson, performed significantly better 
than in other learning scenarios used with learning objects. Freebody et al. (2007) reported 
significantly higher performance gains in middle school classrooms where learning objects were 
used to teach mathematics over a six week period versus "business as usual" classrooms over 
the same time period. Reimer & Moyer (2005) noted that students using virtual manipulatives 
in their study improved significantly in conceptual knowledge, but not in procedural knowledge. 
Finally, Bower (2005) observed that performance gains were contingent upon receiving 
feedback that compared personal results with the group. 



Methodological Issues 


This study reviewed eight articles looking at the use of learning objects in either elementary or 
middle school classrooms. It is commendable that most of these studies emphasized student 
performance, a pattern that has not been observed at higher levels of education. That said, a 
number of challenges remain with respect to improving the analysis of learning objects, 
particularly in middle schools. 

First, while there is good data on how teacher use learning objects, limited information is 
provided on teachers' attitudes toward learning objects. Only three studies examined the 
teacher's perspective (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007). 

Second, even though a wide range of learning objects exist, the majority of papers focus on a 
single learning object. It is difficult to determine whether the evaluation tools used in one study 
generalize to the full range of learning objects that are available. 

Third, sample populations tested in many studies are relatively small and poorly described 
making it challenging to extend any conclusions to a larger population. All but two studies 
(Bower, 2005; Freebody et al., 2007) examined less than 70 students. 

Fourth, triangulation of data collection is somewhat limited with only two papers using more 
than two data collections procedures (Freebody et al., 2007; Reimer & Moyer, 2005). 
Furthermore, only one unpublished report (Freebody et al., 2007) looked at student attitudes, 
teacher attitudes, and student performance simultaneously. 

Finally, while most evaluation studies reported that students benefited from using learning 
objects, the evidence is based on loosely designed assessment tools with no validity or 
reliability. Only two studies reviewed offered estimates of reliability (Kong & Kwok, 2005; Liu & 
Bera, 2005) and no studies provided validity data. The lack of reliability and validity of 
evaluation tools reduces confidence in the results presented to date. 

In summary, previous methods used to evaluate learning are limited with respect to examining 
multiple perspectives of learning objects, particularly those of teachers, number of learning 
objects assessed, sample size, reliability, and validity. 

Purpose 

The purpose of this study was to examine the impact of learning objects in middle school 
classrooms from the perspective of both teachers and students. 

Method 

Key Research Questions 

In order to examine the impact of learning objects on middle school students, the following 
questions were addressed in the data analysis: 

1. How do teachers use learning objects in their classrooms? (descriptive analysis of teacher use 
questions); 

2. How do teachers rate learning, quality, and engagement of learning objects? (descriptive analysis of 
teacher survey - LOES-T in Appendix A); 

3. What was the overall impact of learning objects according to teachers? (analysis of qualitative 



teacher comments - Question 9 in Appendix A); 

4. How do students rate learning, quality, and engagement of learning objects? (descriptive analysis of 
student survey - LOES-S in Appendix B); 

5. What do students like and dislike most about learning objects? (qualitative analysis of student 
comments - Questions 13 and 14 in Appendix A); and 

6. How do learning objects affect student performance (t-test comparing pre and post scores)? 

Addressing Previous Methodological Concerns 

In order to address the methodological challenges noted in the literature review, the following 
steps were taken: 

1. a relatively large, diverse, sample was used; 

2. reliability and valid surveys were used ; 

3. both teacher and student perspective were assessed; 

4. a measure of student performance was included; and 

5. 10 different learning objects in mathematics and science were tested. 

Sample 

Teachers. The teacher sample consisted of 8 teachers (3 males, 5 females; Grades 4 to 8) and 
14 classrooms (a number of teacher used learning objects more than once). Teaching 
experience ranged from 0.5 to 22 years with a mean of 6.6 (SD = 7.9). Subject areas taught 
were math (n=12 classrooms) and science (n = 2 classrooms). A majority of the teachers rated 
their ability to use computers as strong or very strong (n = 5) and their attitude toward using 
computers as positive or very positive (n = 7). In spite of the high ability and positive attitude, 
none of teachers used computers in their classrooms more than once a month. 

Students. The student sample consisted of 262 middle school students (143 males, 119 
females), 10 to 15 years of age (M = 12.5, SD = 1.3). Eighty-four percent (n = 231) of the 
student reported being good at working with computers, 90% (n = 235) liked working with 
computers at school, and 87% (n = 229) noted that they felt comfortable working with 
computers in school. 

The population base spanned two separate boards of education, seven middle schools, and 
eight different classrooms. The students were selected through convenience sampling and had 
to obtain signed parental permission to participate. 

Learning Objects. A total of 10 unique learning objects were selected by middle school 
teachers in this study (see http://faculty.uoit.ca/kay/papers/ MSLO.html for a complete set of 
links to the learning objects used). Most of the learning objects covered topics in mathematics 
including factoring, estimating, triangular pyramids, multiplying fractions, geometric facts, 
probability, and measuring similar shapes. Two science learning objects covered rocks and soil 
and volume and displacement. 

Procedure 

Teachers from three boards of education in Ontario, Canada were contacted via email and 
informed of the learning object study by an educational coordinator. Participation was voluntary 
and a participant could withdraw from the study at any time. Each teacher received a half day 



of training in November on how to choose, use, and assess learning objects (see 
http://www.education.uoit.ca/lordec/lo_use.html for more details on the training provided). 
They were then asked to use at least one learning object in their classrooms by April of the 
following year. Email support was available throughout the duration of the study. All students in 
a given teacher's class used the learning object that the teacher selected. However, only those 
students with signed parental permission forms were permitted to fill in an anonymous, online 
survey about their use of the learning object. In addition, students completed a pre- and post¬ 
test based on the content of the learning object. 

All teachers in this study used learning objects in a lab setting. In order to simulate real middle 
school environment as much as possible, teachers were given full control over the learning 
object they selected, the purpose for using the learning object, how long the learning object 
was used, teaching strategies for using the learning objects, and the design of the pre- and 
post-tests. 

Data Sources 

Teacher Use. Teachers were asked (a) how much preparation was involved when using 
learning objects (e.g., how long it took them to find and integrate learning objects into their 
classroom), (b) their purpose for using the learning object (e.g., motivate students, teach a 
new concept, review, supplementing a lesson), (c) strategies they used to integrate learning 
objects (e.g., demonstration, providing a set of guiding questions, let student explore, 
discussion after learning object) and (d) how long the learning object was used in their 
classroom. 

Teacher survey. After using a learning object, each teacher completed the Learning Object 
Evaluation Scale for Teachers (LOES-T) to determine their perception of (a) how much their 
students learned (learning construct), (b) the quality of the learning object (quality construct), 
and (c) how much their students were engaged with the learning object (engagement 
construct). Data from the LOES-T showed low to moderate reliability (0.63 for learning 
construct, 0.69 for learning object quality construct, 0.84 for engagement construct), and good 
construct validity using a principal components factor analysis. See Kay & Knaack (2007c) for a 
detailed of the teacher-based learning object scale. 

Teacher comments. Finally, teachers were asked to comment on the overall impact that the 
learning object had on learning (Q9, Appendix A). 


Table 1 . Coding Scheme to Categorize Student Comments about Learning Objects 



Learning 

Category Label 

Criteria 

Challenge 

Refers to the ease/difficulty of the concepts being 
covered. Basically whether the content level of the LO 
matched the student’s cognitive level /understanding. 

Code “it was easy” in here, but not “it was easy to use”. 

Learn 

Student comments about a specific or general 
leaming/teaching issue involved in using the LO. 

Visual 

The student mention as visual feature of the LO that 
helped/inhibited their learning. 

Engagement 

Category Label 

Criteria 

Compare 

Student compares LO to another method of learning. 

Engage 

Student refers to program as being OR not being 
fun/enjoyable/engaging/interesting. 

Technology 

The student mention a technological issue with respect to 
using the LO. 

Quality 

Category Label 

Criteria 

Animate 

Refers to quality of animations /moving pictures. 

Audio 

Refers to some audio/sound aspect of the learning object. 

Easy 

Refers to clarity of instructions or how easy/hard the LO 
was to use. It does not refer to how easy/hard the concept 
was to learn. 

Graphics 

Refers to static picture or look of the program (e.g., 
colours). 

Help 

Refers specifically to help/ hints/ instructions/ feedback 
provided by the LO. 

Interactive 

Student refers to some interactive part feature of the LO. 

Control 

Refers to student control of choice/pace in using the LO. 

Organization/Design 

Refers to quality of organization/design or the LO. 

Text 

Refers to quality/amount of text in LO. 

Theme 

Refers to overall/general theme or CONTENT of LO. 


Student comments. Students were asked to comment on what they liked and disliked about 
the learning object (Appendix B - questions 13 and 14). These qualitative items were organized 
according to the three main constructs identified in the literature review (learning, quality, and 
engagement) and analysed using the coding scheme provided in Table 1. This coding scheme 
(Kay & Knaack, 2007a) was used to categorize 596 student comments. Each comment was 
then rated on a five-point Likert scale (-2 = very negative, -1 = negative, 0 = neutral, 1 = 





























positive, 2 = very positive). Two raters independently assessed all comments made by 
students based on categories listed in Table 1 and the 5 point rating system. The approximate 
inter-rated reliability achieved was 75% for categories and 80% for ratings. The two raters then 
met to discuss any discrepancies. Inter-rater reliability increased to 99 % for categories and 
100% for ratings. 

Note that the total impact of any one category was determined by multiplying the mean rating 
by the total number of students who made a comment. For example, from Table 4, the impact 
of visual supports on learning was calculated by multiplying the mean which was 1.00 by the 
number of student s who commented about visual supports (15) for a total of 15.0. 

Student performance. Students completed a pre-test and post-test created by each teacher 
based on the content of the learning object used in class. Questions for pre- and post-test were 
identical in form, but differed in the raw numbers used. The type of questions asked varied 
according to the goal of the specific learning objects. Some tests focussed primarily on factual 
knowledge while others assess higher order thinking focussing on "what-if" scenarios. The 
measure was used to determine student performance. Because of the wide range of learning 
objects used, it was not possible to assess the validity of this test data. 

Results 

Use of Learning Objects 

Finding a learning object. Thirty-six percent (n = 5) of the teachers reported that finding an 
appropriate learning object took them less than 30 minutes. Forty-three percent (n = 6) took 30 
to 60 minutes. And the remaining 21% (n = 3) took over an hour. 

Preparing a learning object lesson. With respect to preparation for using the learning object 
in class, 14 % of the teachers (n = 2) spent little or no time, 36% (n = 5) spent less than 30 
minutes, 36% (n = 5) spent 30 to 60 minutes, and the remaining 14% (n = 2) spent over an hour. 

Using a learning object. On average, teachers used learning objects for 26.1 minutes (SD 
11.5) with a range of 15 to 60 minutes. Students worked on their own on computers in all 
classrooms. 

Reason for using a learning object. The top three reasons cited by teachers for using 
learning objects were to motivate students about a topic (64%, n = 9), provide another way to 
look at a concept (57%, n = 8), and review a previous concept (50%, n = 7). About one third of 
the time, teachers used learning objects to teach a new concept (36%, n = 5) or introduce a 
new topic (29%, n=4). Teachers rarely or never used learning objects for homework or 
extending a concept. 

Strategies for using learning objects. Teachers in this study typically provided a brief 
introduction to a learning object then let the students start exploring on their own (74%, 
n=10). Only three teachers (21%) offered a formal demonstration of the learning object before 
the class used it. Forty-three percent of teachers (n = 6) chose to have a class discussion about 
the learning object after it was used by students. Only one teacher prepared a formal handout 
with guiding questions to be use by students while using the learning object. 



Teacher Rating of Learning Objects 


Learning. The mean rating for impact on learning (Items 1 and 2 - Appendix A) was 12.1 (SD 
= 2.2) or 6.1 on a 7-point scale. This indicates that most teachers agreed that the learning 
object had a positive impact on student learning (Table 2). 

Table 2. Teacher Rating of Learning, Quality, and Engagement for Learning Objects 


Scale 

No. 

Items 

Possible Range 

Actual 

Range 

Observed 

Mean (S J)) 

Learn 

2 

2 to 14 

6 to 14 

12.1 (2.2) 

Quality 

3 

3 to 21 

13 to 21 

17.3(1.9) 

Engagement 

3 

2 to 21 

18 to 21 

19.6(1.2) 


Quality of Learning Object. The mean rating of learning object quality (Items 3 to 5 - 
Appendix A) was 17.3 (SD = 1.9) or 5.8 on a 7-point scale. Most teachers agreed or strongly 
agreed the learning object was of good quality (Table 2). 

Engagement. Teachers also rated engagement of learning objects (Items 6 to 8 - Appendix A) 
quite high with a mean score of 19.6 (SD = 1.2) or 6.5 on a 7-point scale. A majority of 
teachers, then, felt students were engaged while using learning objects. The range of learning 
object engagement scores was quite narrow (18 to 21) providing further support for the 
conclusion that teachers felt that the learning objects they selected were engaging (Table 2). 

Teacher comments about learning objects. Two main themes emerged from the 22 
comments teachers made about the overall impact of learning objects: learning and 
engagement. With respect to learning, 10 teachers commented on how learning objects kept 
students on task, helped review for a topic, or provided visual cues to help students learn. 

All students were on task during the 90 minute lesson (note that math is usually only a 60 minute class! 

It provided immediate feedback to the students and offered excellent re-enforcement of the lesson. 

I believe [the learning object] helped my visual-spatial and hands-on learners. 

Eight teachers commented that the learning objects were motivating or engaging for their students. 

The learning objects added a motivating tool, and offered a novel way of introducing a concept. 

It not only helped review concepts, but it also motivated students about probability (the topic of our new 
unit). 

The learning object helped to motivate students and get them excited about learning math. 

Student Rating of Learning Objects 

Learning. Students rated learning objects lower than teachers with respect to learning (Items 
1 to 5 - Appendix B) (M = 16.9, SD = 4.9) with a mean item rating of 3.4 out 5 (or 4.8 out of 7). 
Students were between neutral and agree with respect to how much they felt the learning 
objects contributed to their learning. The range of scores was broad (5 to 25) indicating that 










there was considerable variability with this construct (Table 3). The mean range was 15 (SD = 
3.6) out of a possible 20 point spread. 

Quality of Learning Objects. Students rated the quality of learning objects (Items 6 to 9 - 
Appendix B) higher than the learning value, although the mean item rating was still lower than 
that of the teachers. The mean item rating was 3.9 out of 5 (5.4 out of 7) indicating that most 
students agreed that the learning objects they used were of good quality. The range of learning 
object quality scores (4 to 20) showed considerable variability (Table 3). The mean range was 
9 (SD = 3.6) out of a possible 16 point spread. 

Engagement. Ratings of learning object engagement (Items 10 to 12 - Appendix B) were 
moderate (M = 10.8, SD = 3.3) with a mean item rating of 3.6 out or 5 (or 5.0 out of 7). In other 
words, as was the case with the learning construct, students were somewhere in between 
neutral and agree when assessing the engagement value of the learning objects they used. 
High variability among student engagement ratings is supported by the wide range of scores 
reported (3 to 15). The mean range was 9 (SD = 1.9) out of a possible 12 point spread. 

Table 3. Description of Student Learning Object Evaluation Scales (LOES-S) 


Scale 

No. Items 

Possible 

Range 

Actual 

Range 

Observed 

Mean (SX>) 

Learn 

5 

5 to 25 

5 to 25 

16.9 (4.9) 

Quality 

4 

4 to 20 

4 to 20 

15.5 (32) 

Engagement 

3 

3 to 15 

3 to 15 

10.8 (2.7) 


Student comments about learning objects. Nearly 600 student comments are summarized 
in Table 4. With respect to learning, the visual support that a learning object offered toward 
understanding a concept was rated the highest, whereas the pedagogical challenge of the 
learning object was rated quite low. In other words, some students liked the visual benefits of 
learning objects, however, more students thought the learning object they used was not 
challenging enough. 

With respect to rating the quality of learning objects, ease of use was the highest rated 
feature, followed by graphics and animation. On the other hand, the quality of help and the 
excessive amount of text were rated the lowest. 

Regarding engagement, the motivational or engaging features of learning objects were rated 
the highest, followed a general positive attitude toward using technology and a preference for 
using learning objects over other teaching methods. 

Student Performance 

Differences between pre- and post-test scores were calculated for classes where the learning 
object was not used for review. This yielded a total of 89 students. Student performance scores 
improved by an average of 40.9% from 30.5% to 71.4%. This change was significant (t = 
-10.59, df = 88, p < .001). The effect size (based on Cohen's d) of 1.55 is considered "large" 










according to Thalheimer & Cook (2002). 


Table 4. Summary of Student Comments about Learning Objects 


Category 

Mean 


SD 

n 

Total 

Effect 






Mean * n 


Learning 

Visual Supports 

1.00 

0.00 

15 

15.0 

Overall Learning 

-0.07 

1.19 

123 

-8.0 

Challenge 

-0.58 

1.07 

85 

-49.0 


Quality 


Easy 

0.92 

0.63 

26 

24.0 

Graphics 

0.67 

0.82 

24 

16.0 

Animation 

0.92 

0.67 

12 

11.0 

Organization 

0.07 

1.07 

27 

2.0 

Theme 

0.03 

1.19 

39 

1/0 

Control 

-0.06 

1.11 

18 

-1.0 

Audio 

-0.25 

1.14 

28 

-7.0 

Text 

-1.00 

1.00 

7 

-7.0 

Help 

-0.29 

1.17 

45 

-13.0 


Engagement 


Engagement / Motivation 

0.53 

1.22 

80 

42.0 

Liking Technology 

0.87 

0.68 

30 

26.0 

Compare with other method 

0.91 

0.87 

22 

20.0 

Interactivity 

0.73 

0.70 

15 

11.0 


The purpose of this study was to examine the impact of learning objects in middle school 
classrooms. Six measures of use and/or effectiveness were used including teacher use, teacher 
ratings, teacher comments about overall impact, student ratings, student comments about 
what they liked and did not like, and student performance. The results from each of these 
measures will be discussed in turn. 

Teacher Use 

Previous research is relatively silent with respect to data on finding and preparing to use 
learning objects; this study provides new information in this area. It appears that searching for 
and planning to use a learning object does not take excessive time - roughly an hour on 
average. Teachers did not comment that time was a problem, so we can assume that this 
aspect of learning object use was not a prominent issue. 










































It is interesting that 50 to 65% of teachers used learning objects to supplement concepts that 
they had already taught. Only one-third used learning objects to help introduce a new topic 
before a formal lesson or to teach a new concept on its own. One could speculate that teachers 
were being somewhat cautious with a new teaching tool in their classroom. 

With respect to using learning objects, most teachers offered a brief introduction and let the 
students explore on their own. This finding is consistent with the behaviour of middle school 
teachers observed in previous research (Liu & Bera, 2005; Nurmi & Jaakola, 2006; Reimer & 
Moyer, 2005). In addition, 40% of teachers chose to have a class discussion after students 
used learning objects. Finally, most teachers did not prepare a formal handout - students were 
left to investigate and draw conclusions on their own. Perhaps teachers felt that, since the 
concept being covered was a review, there was no need to provide additional support, 
especially when a formal discussion was common practice after students use. It might also 
mean that teachers felt that the learning objects they selected should stand on their own in 
terms of teaching. 

Teacher Ratings and Comments (Learning, Quality, and Engagement) 

It is reasonable to conclude that teachers felt that the learning objects they selected were 
good quality, engaging tools that supported learning. Ratings were very high, often 6 on a 7- 
point Likert scale. On the one hand, it is somewhat predictable that teachers would rate 
learning objects high - after all they did select them. On the other hand, teachers rated these 
learning objects after they watched them being used by students in their classroom. Positive 
reaction from teachers in this study is consistent with previous findings for middle school 
teachers using learning objects (Clarke & Bowe, 2006a, 2006b). 

Teacher Comments 

With respect to learning, teacher comments supported the survey results. Many teachers felt 
their students were on task and that the learning objects offered a good review of concepts. In 
addition, learning objects were thought to be engaging. These comments reflect results 
reported in previous research (Clarke & Bowe, 2006a, 2006b). It should be noted that these 
comments reflect the opinions of a relatively small number of middle school teachers and that 
more research is needed to build and extend upon or replicate these findings. 

Student Ratings (Learning, Quality, and Engagement) 

Students, on average, "slightly agreed" that the learning objects they used were good quality, 
engaging learning tools that helped them learn. The students in this study appreciated learning 
objects less than in other reports (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007). Part of 
the difference might be attributed to short- versus long-term use. Learning objects in the 
current study were used less than 30 minutes on average compared to extensive six weeks of 
use reported by the Learning Federation (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007). 
Presumably, students can experience a wider range of learning objects and strategies over a 
longer time period and teachers can become more proficient at integrating them into the 
curriculum. It could also be speculated that the reason why student ratings were lower in this 



study was that learning objects developed by the Learning Federation were of higher quality. 
An argument could be made that a profitable next step in evaluating learning objects is to 
examine the specific features of learning objects that students like, coupled with the 
pedagogical strategies used. 

It is worthwhile noting that the range of student scores was broad for all three constructs. In 
addition, the mean rating of student comments on "Overall Learning" was essentially zero. For 
any given learning object, some students liked it a lot; others disliked it a lot, even when it 
was the same learning object and the same teacher. Learning objects, in spite of the many 
potential benefits, may not be an effective learning medium for some students. The challenge 
that remains is to understand the nature of individual differences that exist. 

Finally, teachers rated learning, quality, and engagement higher than students. This 
discrepancy has not been observed in previous research, although it could be a potential 
problem. What middle school teachers think is an effective, high quality, motivating teaching 
tool, may not correspond with what their students actually experience. It may also reflect the 
fact that teachers are so busy trying to organize and run a class that is difficult for them to 
accurately assess specific and overall impact with respect to learning objects. One suggestion 
for future research is to include observational data by a third party. 

Student Comments about Learning Objects 

Student comments offer a more detailed look at what students liked and did not like when 
using learning objects. Students preferred learning objects that were easy to use and had good 
interactivity, visual supports, animations, and graphics. They did not like learning objects that 
were not challenging enough, nor did they like inadequate help features and excessive amounts 
of text. These comments are similar to those made by students in previous studies (Clarke & 
Bowe, 2006a, 2006b; Freebody et al., 2007; Reimer & Moyer, 2005). 

Student Performance 

It is clear that learning performance increased when learning objects were used. A mean jump 
of over 40% resulted in a large effect size. This result was observed repeatedly in previous 
research (Akpinar & Bal, 2006; Bower, 2005; Kong & Kwok, 2005, Liu & Bera, 2005; Nurmi & 
Jaakkola, 2006; Reimer & Moyer, 2005). The data also suggest that teachers' assessment of 
learning impact in this study is more closely aligned to performance that than students 
assessment of learning. Recall that students were relatively moderate when rating the learning 
impact of learning objects. These modest ratings did not match significant jumps in 
performance. 

The inconsistency between increases in student performance and essentially neutral or slightly 
positive student attitudes is somewhat of a mystery. One would predict that attitude and 
performance would be more closely aligned as in the Freebody et al. report (2007). One might 
expect significant gains in performance for most teaching methods, simply because the 
concepts being taught are new. Pre-tests scores are predictably low because students have 
little or no knowledge of the concepts to be taught. It is not clear from the design of this study, 
whether similar gains would have occurred with another teaching approach. An experimental 



design, like the one used by Freebody et al. (2007) is needed to establish a stronger causal 
link. 

Another issue worth noting is the possibility of a Hawthorne effect. Students may have 
improved in performance due to the fact that "something new" was introduced in the 
classroom. However, this explanation is somewhat unlikely for two reasons. First, only 22 out 
of 596 comments made about learning objects referred to learning objects perceived as 
something different. Second, roughly 90% of the sample population reported that they felt 
comfortable with computers. Students in this study, then, may not have seen the use of 
learning objects as "something new". Investigation of long term learning object use is 
necessary to establish the validity of a Hawthorne effect. 

It is also important to note that improvement in student performance does not mean that 
learning objects were solely responsible for gains observed, regardless of a possible Hawthorne 
effect. As in previous studies (Clarke & Bowe, 2006a, 2006b), a number of teachers used these 
tools in combination with a more formal lessons or class discussion, so the influence of learning 
objects is partially confounded by additional teaching techniques. It is reasonable, and perhaps 
desirable, that learning objects are integrated within a full classroom lesson that involves 
multiple teaching strategies. 

Implications for Education 

There are several implications for middle school educators who intend to use learning objects in 
their classrooms. First, it will take on average between one and two hours to find and prepare 
to use learning objects. Second, both teachers and students are positive about the use of 
learning objects in the classroom, but the impact of learning objects may vary greatly within 
the same classroom. Accommodations may have to be made for students with different ability 
and interest levels. Third, based on the behaviour of most teachers in this study, combining the 
use of learning objects with a formal lesson using a brief introduction and or class discussion 
appears to work well. 

Caveats and Future Research 

Considerable effort was made to ensure the reliability and validity of the results reported in this 
study, yet several limitations exist which provide opportunities for future research endeavours. 
First, even though the student population was relatively large and balanced in terms of gender, 
topics covered were restricted to mathematics and science. Different results may be observed 
with other subject areas. In addition, only eight teachers were involved in this study - a future 
larger scale investigation is warranted to increase generalizability. 

Second, while the overall impact of learning-object lessons on student performance was large, 
the impact of specific instructional strategies was not examined. In other words, we don't know 
which teaching strategies work best with learning objects. Perhaps different strategies work 
with different kinds of learning objects. 


Third, short term use of learning objects was assessed in this study. It is reasonable to follow 
the pathway of the researchers (Clarke & Bowe, 2006a, 2006b; Freebody, et al., 2007) 



investigating learning objects created by the Learning Federation (see 
http://www.thelearningfederation.edu.au/default.asp). However, access to technology and 
learning objects is far more limited in many countries where well developed, concentrated 
collections of learning objects are not readily available. 

Finally, the study was designed to look at the overall impact of learning objects - the impact of 
specific kinds of learning objects was not looked at. It is possible that certain categories of 
learning objects may have decidedly different impacts on learning. For example, a tool-based 
learning object (e.g., http://www.articlel9.com/shockwave/oz.htm) may be good for exploring 
a new concept where as a question-and-answer learning object (e.g., http://amazing- 
space.stsci.edu/resources/explorations/trading/directions.html ) may be best for review. 
Developing and evaluating a classification system for learning objects could be an important 
next step in learning object research. 

Conclusions 

The results of this study support and add new information to the relatively small knowledge 
base that exists with respect to the use of learning objects in middle school. Perhaps the 
biggest contribution was methodological rigour. Three forms of data collection were combined 
to analyse the impact of learning objects: teacher attitude, student attitude, and student 
performance. In addition a relatively large, diverse sample of students and multiple teachers 
were assessed using a number of different learning objects. Finally, reliable and valid measures 
were used. This comprehensive approach to evaluation is rarely observed in learning object 
research. The positive results reported are consistent with past studies examining student 
attitudes, teacher attitudes, and student performance with respect to learning objects. 

Previous researchers have focussed on student performance and, to a lesser extent, student 
attitudes about learning objects. Relatively little is known about the teacher's viewpoint. This 
paper looked at eight teachers who use learning objects in 14 middle school classrooms. They 
were very positive about the learning value, quality, and engagement of learning objects and 
these attitudes were consistent with significant gains in learning performance. In fact, the 
teacher impressions were more closely aligned with student performance than those of 
students. 

This study also provides some insight into how teachers prepare for and use learning objects, 
an area that has not been examined before. Most middle school teachers take anywhere from 
one to two hours to produce a lesson that is often focussed on review, and to a lesser extent 
introducing new concepts. 
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Appendix A - Learning Object Evaluation Scale for Teachers 

1 - Strongly Disagree / 2 - Disagree / 3 - Slightly Disagree / 4 - Neutral / 

5 - Slightly Agree / 6 - Agree / 7 - Strongly Agree 



1 

2 

3 

4 

5 

6 

7 

Learning 

1. The graphics and 
animations from the 
learning object helped 
students leam. 

1 

2 

3 

4 

5 

6 

7 

2. The students were 

able to leam from the 
learning object. 

1 

2 

3 

4 

5 

6 

7 

Quality 

3. The learning object 
was easy for students 

to use. 

1 

2 

3 

4 

5 

6 

7 

4. The learning object 
was easy to leam. 

1 

2 

3 

4 

5 

6 

7 

5. The students found the 
learning object 
instructions clear. 

1 

2 

3 

4 

5 

6 

7 

Engagement 

6. The students liked 
interacting with the 
learning object. 

1 

2 

3 

4 

5 

6 

7 

7. The students were on 
task while using the 
learning object. 

1 

2 

3 

4 

5 

6 

7 

8. Students were 

motivated while using 
the learning object. 

1 

2 

3 

4 

5 

6 

7 


What was the overall impact of the learning object on your lesson? 










Appendix B - Learning Object Evaluation Survey - Students 

1 - Strongly Disagree / 2 - Disagree / 3 - Neutral / 4 - Agree / 5 - Strongly Agree 



1 

2 

3 

4 

5 

Learning 

1. Working with the learning 
object helped me learn. 

1 

2 

3 

4 

5 

2. The feedback from the 
learning object helped me 
learn. 

1 

2 

3 

4 

5 

3. The graphics and 
animations from the 
learning object helped me 
learn. 

1 

2 

3 

4 

5 

4. The learning object helped 
teach me a new concept. 

1 

2 

3 

4 

5 

5. Overall, the learning 
object helped me learn. 

1 

2 

3 

4 

5 

Quality 

6. The help features in the 
learning object were 
useful. 

1 

2 

3 

4 

5 

7. The instructions in the 
learning object were easy 
to follow. 

1 

2 

3 

4 

5 

8. The learning object was 
easy to use. 

1 

2 

3 

4 

5 

9. The learning object was 
well organized. 

1 

2 

3 

4 

5 

Engagement 

10. I liked the overall theme 
of the learning object. 

1 

2 

3 

4 

5 

11. I found the learning 
object motivating. 

1 

2 

3 

4 

5 

12. I would like to use the 
learning object again. 

1 

2 

3 

4 

5 


What, if anything, did you LIKE about the learning object? 
What, if anything, did you NOT LIKE about the learning object? 












